Image display device, method for driving the same, and computer readable recording medium

ABSTRACT

An image display device, a method for driving the same, and a computer readable recording medium are provided. The image display device includes a speech acquirer configured to acquire a speech query associated with a query created by a user, a display configured to display a query list composed of candidate queries having the same as or similar semantic as the acquired speech query, and an operation performer configured to perform an operation related to the query selected from the displayed query list.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 14/854,759, filed on Sep. 15, 2015, which claimed priority under 35U.S.C. § 119(a) of a Korean patent application filed on Sep. 16, 2014 inthe Korean Intellectual Property Office and assigned Ser. No.10-2014-0122871, the entire disclosure of which is hereby incorporatedby reference.

TECHNICAL FIELD

The present disclosure relates to an image display device, a method fordriving the same, and a computer readable recording medium. Moreparticularly, the present disclosure relates to an image display device,which can present candidate queries related to a query that is createdby a user in a device, such as a television (TV) or a portable phone,and if the user selects a desired query, provide a search for theselected query or answer information, a method for driving the same, anda computer readable recording medium.

BACKGROUND

Speech recognition identifies linguistic semantic content from speech.For example, speech recognition corresponds to a process of identifyinga word or a word string through an input of speech waveforms andextracting semantics associated with the identified word or word string.Different techniques may be used on a speech waveform to identify wordsor word strings. In an exemplary embodiment, the different speechrecognition techniques may be classified into five processes includingspeech analysis, phoneme recognition, word recognition, sentenceinterpretation, and semantic extraction. However, performing all of thedifferent techniques on a speech waveform is undesirably complex.Therefore, conventional speech recognition methods generally include atleast one of the processes of speech analysis, phoneme recognition, andword recognition.

Previously, the hardware necessary to implement a human-machine speechrecognition interface has undesirably limited the use of speechrecognition. However, with the development of large-scale integratedcircuits, a speech recognition device and a voice synthesis device canbe realized on an integrated circuit having a size of severalmillimeters in width and length, and thus allows a speech input/outputdevice to be more easily implemented.

In an exemplary embodiment, a speech input/output device may be used toreceive a bank balance inquiry, a stock price inquiry, an application ofcommunication sales, a credit card inquiry, a hotel or flight seatreservation, etc. via phone. However, a common problem associated withgeneral inquiries using natural language based speech question andanswer (Q/A) services, such as S-Voi** or Si-*, is that there are toomany questions that are unable to be understood. Further, when the Q/Aservices are implemented using cloud computing techniques, an increasein response delays with respect to a natural language query introducesundesirable network interruptions. When network interruptions occur, itis not possible for the device to perform additional speech recognitionQ/A services.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide an image display device, which can presentcandidate queries related to a query that is created by a user in adevice, such as a television (TV) or a portable phone, and if the userselects a desired query, provide a search for the selected query oranswer information, a method for driving the same, and a computerreadable recording medium.

In accordance with an aspect of the present disclosure, an image displaydevice is provided. The image display device includes a speech acquirerconfigured to acquire a speech query associated with a query created bya user, a display configured to display a query list composed ofcandidate queries having the same or similar semantic as the acquiredspeech query, and an operation performer configured to perform anoperation related to the query that is selected from the displayed querylist.

In accordance with another aspect of the present disclosure, a methodfor driving an image display device is provided. The method includesacquiring a speech query associated with a query created by a user,displaying a query list composed of candidate queries having the same orsimilar semantic as or to a semantic of the acquired speech query, andperforming an operation related to the query selected from the displayedquery list.

In accordance with another aspect of the present disclosure, anon-transitory computer readable recording medium is provided. Thenon-transitory computer readable recording medium includes a program forexecuting a method for driving an image display device, wherein themethod for driving an image display device includes acquiring a speechquery associated with a query created by a user, generating a query listthat is composed of candidate queries having the same or similarsemantic as a semantic of the acquired speech query, and providingsearch information or answer information for the query that is selectedfrom the query list.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram illustrating a speech service system according to anembodiment of the present disclosure;

FIG. 2 is a diagram explaining a speech service according to anembodiment of the present disclosure;

FIG. 3 is a block diagram illustrating a structure of an image displaydevice of FIG. 1 according to an embodiment of the present disclosure;

FIG. 4 is a block diagram illustrating another structure of an imagedisplay device of FIG. 1 according to an embodiment of the presentdisclosure;

FIG. 5 is a diagram explaining an operation of a semantic informationgenerator of FIG. 4 according to an embodiment of the presentdisclosure;

FIG. 6 is a block diagram illustrating a detailed structure of asemantic information generator of FIGS. 4 and 5 according to anembodiment of the present disclosure;

FIG. 7 is a flowchart illustrating a process of driving an image displaydevice according to an embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a process of driving an image displaydevice according to another embodiment of the present disclosure;

FIGS. 9A, 9B, and 9C are diagrams illustrating a screen display methodof an image display device according to an embodiment of the presentdisclosure;

FIG. 10 is a flowchart illustrating a process of driving an imagedisplay device according to an embodiment of the present disclosure;

FIG. 11 is a diagram explaining a process of searching for a similarityquery according to an embodiment of the present disclosure;

FIG. 12 is a block diagram illustrating another structure of an imagedisplay device of FIG. 1 according to an embodiment of the presentdisclosure; and

FIG. 13 is a block diagram illustrating the structure of a serviceproviding device illustrated in FIG. 1 according to an embodiment of thepresent disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 1 is a diagram illustrating a speech service system according to anembodiment of the present disclosure, and FIG. 2 is a diagram explaininga speech service according to an embodiment of the present disclosure.

Referring to FIG. 1, a speech service system 90 according to anembodiment of the present disclosure may include a part or the whole ofan image display device 100, a communication network 110, and a serviceproviding device 120.

The term “include a part or the whole” means that some constituentelements, such as the communication network 110 and the serviceproviding device 120, are omitted and the image display device 100 cansolely operate. For sufficient understanding of the present disclosure,explanation will be made on the assumption that the speech servicesystem 90 includes the whole constituent elements.

The image display device 100 according to an embodiment of the presentdisclosure includes, for example, a television (TV), a display panelsuch as an organic light emitting diode (OLED), liquid crystal display(LCD), a plasma display panel, etc., a portable phone, a portable DVDplayer, a Moving Picture Experts Group phase 1 or phase 2 (MPEG-1 orMPEG-2) audio layer 3 (MP3) device, a tablet personal computer (PC), ora laptop and desktop computer, which can display an image. In order toincrease the response rate, speed, and availability of a naturallanguage question and answer (Q/A) service, the image display device 100may perform two types of operations. One operation constructsinformation in order to provide candidate queries for a query created bya user and the other operation performs rapid information processingthrough recognition of the query created by the user based on theconstructed information.

For example, the image display device 100 may receive query informationfrom the service providing device 120 as the result of recognitionassociated with a user's initial query. The query information may beinformation having the same or similar semantic as or to the semantic ofthe result of recognition. Further, the image display device 100 mayreceive additional query information as the result of recognition thatthe service providing device 120 provides with respect to a query havingthe same semantic as the semantic of the initial query although theexpression type of the initial query may be different from theexpression type of the subsequent query. The image display device 100can generate or construct information for candidate queries usingvarious portions of query information that may have the same semanticalthough the expression types are different from each other. Thereafter,even if a user creates various types of queries, for example, “movie”,“Take me to a movie”, and “Recommend an amusing movie”, the imagedisplay device 100 may search for candidate queries having the same orsimilar semantic as or to the semantic of the query and provide thesearched candidate queries in the form of a list. In this case, thequery may be provided in various forms such as a word, a sentence, or adomain. In the case of a domain, when a user mentions only a domain withrespect to the query, domain queries of the previous record in thedomain related storage, for example, a cache, or a top-k that exists inthe service providing device 120 can be directly recommended. In anexemplary embodiment, the top-k is a query for obtaining k objects thata user desires most.

Referring to FIG. 2, when a user intends to obtain information about amovie, different types of queries, such as “movie” 200 and “Let me knowan amusing movie” 200, may be created (or uttered) by the user. Even ifthe user creates different types of queries, the image display device100 may derive a common semantic from them and present candidate querieson the display device 100 in order to determine the user's query moreaccurately and clearly on based on the derived semantic. In this case,the candidate queries should commonly include the derived semantic. Forexample, if two semantics are derived, the candidate queries shouldinclude only two semantics. If the intention is not accuratelydetermined as shown in FIG. 2 or in order to grasp the intention moreaccurately, the image display device 100 may present “I will show youmovie ranking” 220. If the user selects this, the image display device100 may provide the result of the search, such as “prepared movieraking” or “movie times” 230. In this process, if separately preparedinformation does not exist, the image display device 100 may display anotification and may provide a query that is set by default or search oranswer information by a default operation. In other words, if separateinformation is not prepared as shown in FIG. 2, the image display device100 may provide an audio notification such as “There is no answer youwant to hear” 215, or “I will show you the movie ranking” 220, and thenmay directly provide the information by displaying the associatedinformation on the display device 100.

Further, when the speech query (or speech waveform, speech signal) isreceived from the user, the image display device 100 may recognize thereceived speech query and may first determine whether any candidatequery stored at the image display device 100 exists based on the resultof the recognition. This is related to the response rate. If there is acandidate query that can be provided to the user as the result ofrecognition, the image display device 100 may provide a list includingthe candidate query. When it is determined that there is not a candidatequery, the image display device 100 may receive candidate queries fromthe service providing device 120 and provide the candidate queries inthe form of a list displayed at the image display device 100.Alternatively, the image display device 100 may display the candidatequeries based on the result of recognition performed by the serviceproviding device 120. However, if an error occurs in the communicationnetwork 110 or if a network device in the communication network 110 andthe service providing device 120 experience excessive loads, the imagedisplay device 100 may smoothly operate, such as using candidate queriesstored at the image display device 100. For example, the image displaydevice 100 can include an engine for speech recognition, i.e., aprogram. Further, the engine may selectively operate according to user'sselection or situation.

Alternatively, Nave* Knowledge*, Yah** Answe** provide services througha search for mass Q/A sets that are constructed in cloud computingnetworks. In order to provide the same service through a mobile device,it is necessary to provide Q/A sets having a scale associated with themobile device. However, due to the resource limitations of the mobiledevice (central processing unit (CPU) or storage space that is requiredto search for mass amount of Q/A sets), it is difficult to construct theQ/A sets associated with a mobile device to have the same scale as a Q/Aset associated with a cloud computing network. Accordingly, in anembodiment of the present disclosure, the Q/A sets associated with amobile device are constructed to maintain the search hit rate that issimilar to that of a Q/A set associated with a cloud computing networksuch that the scale of the Q/A sets associated with a mobile device arereduced to suit the mobile device.

For example, since the Q/A that the user can use may differ according tothe time and place, the image display device 100, more accurately, amobile terminal, may change the Q/A set configuration according to thetime and place in consideration of user's situation, a profile, and apast Q/A usage pattern. For example, since a user frequently asks aweather question in the morning, a weather Q/A is included in the set.However, in the afternoon, the weather Q/A may be excluded from the Q/Aset. Further, if a user is in the vicinity of a movie theater, movietime Q/A is included in the set, but is excluded from the set in otherregions. Further, the result of a search when the user asks a“neighboring restaurant search” question in a first location may differfrom the result of search when the user makes a “neighboring restaurantsearch” in a different location.

The communication network 110 includes both wired and wirelesscommunication networks. The wired network may include the Internet, suchas a cable network or a public switched telephone network (PSTN), andthe wireless communication network may include code division multipleaccess (CDMA), wireless CDMA (WCDMA), global system for mobilecommunications (GSM), evolved packet core (EPC), long term evolution(LTE), and wibro. However, the communication network 110 according to anembodiment of the present disclosure is not limited thereto, but may beused, for example, in a cloud computing network under cloud computingenvironment such as a connection network of the next-generation mobilecommunication system to be implemented in future. For example, if thecommunication network 110 is a wired communication network, an accesspoint in the communication network 100 can connect to an exchange of atelephone office, whereas in the case of a wireless communicationnetwork, the access point may connect to a serving general packet radioservice (GPRS) support node (SGSN) or gateway GPRS support node (GGSN)operated by a communication company to process data, or may connect tovarious repeaters, such as base transceiver station (BTS), NodeB, ande-NodeB to process data.

The communication network 110 may include an access point. The accesspoint includes a small base station, such as a femto or pico basestation, which is mainly installed in a building. The femto or pico basestation may be classified depending on a maximum number image displaydevices 100 that can establish communication with the femto or pico basestation according to classification of a small base station. The accesspoint includes a near field communication module for performing nearfield communication, such as ZigBee or Wi-Fi, with the image displaydevice 100. The access point may use transmission controlprotocol/Internet protocol (TCP/IP) or real-time streaming protocol(RTSP) to perform wireless communication. Near field communication maybe performed in various standards, such as radio frequency (RF) andultra wideband (UWB) communication, such as Bluetooth (BT), ZigBee,infrared rays (IrDA), ultra high frequency (UHF), and very highfrequency (VHF). Accordingly, the access point may extract the positionof a data packet, designate the best communication path for theextracted position, and transfer the data packet to a next device, forexample, the image display device 100 along the designated communicationpath. The access point can share several lines in a general networkenvironment, and may include, for example, a router, a repeater, and arelay.

The service providing device 120 may operate as cloud server, and may becalled an external device or a recognition result providing device. Inan exemplary embodiment, the result of recognition may include providinginformation associated with candidate queries beyond providingtext-based information for a speech query. The service providing device120 may include a free creation engine and by executing the freecreation engine, the service providing device 120 recognizes a speechsignal that is provided from the image display device 100 and providesthe result of the recognition to the image display device 100 in a textbased manner. In an exemplary embodiment, the free creation engine maybe a type of speech recognition engine that can provide the result ofrecognition more accurately as a high performance engine in comparisonto a fixed creation engine of the image display device 100.

One of ordinary skill in the art would recognize that the serviceproviding device 120 according to an embodiment of the presentdisclosure is not limited to a cloud server. For example, in the casewhere the communication network 110 is omitted and the image displaydevice 100 performs direct communication with the service providingdevice 120, the service providing device 120 may be an external device,i.e., an access point, or a peripheral device, such as a desktopcomputer. In other words, any type of service providing device 120 canbe used so far as the service providing device 120 provides the resultof recognition for the speech signal that is provided by the imagedisplay device 100 as well as information associated with candidatequeries. The service providing device 120 may be a recognition resultproviding device.

In operation, in order to increase the response rate of the naturallanguage Q/A service, the image display device 100 according to anembodiment of the present disclosure searches for candidate queries thatare similar to the user's query among the Q/A that responded in thepast, and responds to the query using the response related to thesearched query other than understanding the natural language andresponding to the user's question. Since the response is providedthrough the Q/A search, in the case where the service is frequently usedand the Q/A set becomes large, the probability to search for the similarQ/A is increased to cause the response rate also increase therebycausing delays. However, in order to provide the search-based Q/Aservice through a mobile terminal such as a portable phone havinglimited resources (e.g., CPU and storage), the Q/A set should not beextended unlimitedly. In order to provide the Q/A service through themobile terminal, the personalized Q/A set may be constructed based onthe user situation, the user profile, and the past Q/A usage pattern.The Q/A service providing in the mobile terminal may have the advantagesof shortening service response time and increase reliability to providethe service even in the case of network interruption.

FIG. 3 is a block diagram illustrating the structure of an image displaydevice of FIG. 1 according to an embodiment of the present disclosure.

Referring to FIG. 3, an image display device 100 according to anembodiment of the present disclosure may include a part or the whole ofa speech acquirer 300, an operation performer 310, and a display 320.

The term “include a part or the whole” means that a part of constituentelements, such as the speech acquirer 300, is omitted, or anotherconstituent element, such as the operation performer 310, is integrallyconfigured. For sufficient understanding of the present disclosure,explanation will be made on the assumption that the image display device100 includes the whole constituent elements.

The speech acquirer 300 may include a microphone. The speech acquirer300 may acquire a speech created by a user through the microphone andmay provide the created speech to the operation performer 310. That is,the speech acquirer 300 generates and provides a speech signal of theacquired speech waveform.

Further, the operation performer 310 may perform the same operation asthe operation of the constituent elements as described above throughexecution of a part or the whole of a program even if the operationalperformer 310 is not discriminated in hardware from the controller (asreferring to FIG. 4), the speech recognizer, the communicationinterface, and the user interface (UI) screen generator. For example,the operation performer 310 is provided with an engine for speechrecognition by itself, and may obtain the result of recognition for thespeech query created by the user through driving of the speechrecognition engine. Further, the operation performer 310 may transmitthe speech signal to an external service providing device 120 and mayreceive the result of recognition from the external service providingdevice 120. Further, the operation performer 310 may selectively use aninternal engine and an external engine to obtain the result of therecognition.

Alternatively, the operation performer 310 may generate semanticinformation through, for example, word division and semantic derivationusing the text-based result of recognition. Further, the operationperformer 310 may acquire candidate queries associated with thegenerated semantic information and generate the acquired candidatequeries in the form of a list to be transferred or shown to the user.For example, the operation performer 310 may provide the generated listto the display 320 such that the generated list is displayed on thedisplay 320. Further, in the case where the image display device 100 isa TV, the operation performer 310 may transfer the list to a user'sportable phone in the form of a push message so as to select a desiredanswer. The operation performer 310 may search for the candidate queriesstored in the storage provided at the image display device 100 or mayreceive the candidate queries from the external service providing device120 in order to acquire the candidate queries.

Thereafter, the operation performer 310 receives information associatedwith the query selected by the user from the list that is displayed onthe user's portable phone or the display 320 and provides search oranswer information for the query to the user. The search or answerinformation may be provided to the display 320 or the search and answerinformation may be transmitted to the user's portable phone.

In addition, the operation performer 310 may include an image processorconfigured to perform an operation function or may replace the operationthrough a program. In other words, the operation performer 310 mayperform various operations, such as receiving and separating avideo/audio signal that is received from an external broadcastingdevice, and scaling the encoded and decoded video/audio signal. In anexemplary embodiment, the operation performer 310 may include a tuner, asignal separator, a decoder, and a scaler in hardware, or may replacetheir operations through execution of a part of the whole of theprogram.

The display 320 displays a list of candidate queries that aretransferred from the operation performer 310 according to an embodimentof the present disclosure. The candidate queries may be the same as orsimilar to the query information of the speech query created by the userand more particularly, may be composed of queries having the samesemantic, i.e., semantic information. For example, a semantic may be akeyword or a new one derived from the keywords. This will be describedin more detail with reference to FIGS. 4 and 5.

FIG. 4 is a block diagram illustrating another structure of an imagedisplay device of FIG. 1 according to an embodiment of the presentdisclosure, and FIG. 5 is a diagram explaining the operation of asemantic information generator of FIG. 4 according to an embodiment ofthe present disclosure.

Referring to FIG. 4, an image display device 100′ according to anembodiment of the present disclosure may include a part or the whole ofa communication interface 400, a speech acquirer 410, a storage device420, a controller 430, a display 440, a UI screen generator 450, and aspeech recognizer 460.

The term “include a part or the whole” means that a part of constituentelements, such as the speech recognizer 460, is omitted or a part ofconstituent elements, such as the storage 420 or the UI screen generator450, is integrally configured with the controller 430. For sufficientunderstanding of the present disclosure, explanation will be made on theassumption that the image display device 100′ includes the wholeconstituent elements.

According to an embodiment of the present disclosure, being aside fromthe processing of image data for image processing, the communicationinterface 400 may transmit a speech signal associated with a speechwaveform that is acquired from the speech acquirer 410 to the externalservice providing device 120 and may receive the result of recognitionfor the speech signal. For example, in the case where the image displaydevice 100′ is configured to use the service providing device 120 toobtain the result of a recognition, the communication interface 400 mayfirst attempt communication with the service providing device 120 toobtain the result of the recognition, and if an error occurs in thecommunication network 110 or if a load becomes heavy at one or morenetwork nodes, the communication interface 400 may notify the controller430. Accordingly, the controller 430 may operate the internal speechrecognizer 460 to obtain the result of recognition.

Further, the communication interface 400 may perform communication withan external device to obtain response or search information associatedwith the query selected by the user from the candidate query list thatis displayed on the display 440. For example, if the user requests amovie ranking through query selection on the list, the communicationinterface 400 may perform an operation to connect to a server of aprofessional search portal site, such as Nave*. In other words, thecommunication interface 400 requests the corresponding information andreceives a result of request. In this process, various operations forconverting the information, such as decoding operation, may beperformed.

Speech acquirer 410 is similar to the speech acquirer 300 as describedabove with reference to FIG. 3 and thus further description is omitted.

The storage 420 may store various portions of data or information thatis processed at the image display device 100′. For example, in anembodiment of the present disclosure, information associated with thecandidate queries may be stored to match the semantic information. Inother words, as shown in FIG. 5, candidate queries Q1 and Q3 are storedto match semantic information W1 and W2. Accordingly, if the semanticinformation W1 and W2 are derived as the result of the query created bythe user, the storage unit 420 may output information on the queries Q1and Q3 that include two semantics W1 and W2. In practice, such a processmay be performed in association with a semantic information generator470.

Referring to FIG. 4, the storage 420 may use the semantic informationgenerator 470 in order to correlate the query information for thecandidate queries with the semantic information to be stored or in orderto derive the semantic information from the query created by the user tosearch for a desired candidate query. In this case, the queryinformation for the candidate queries may be constructed using theresults of a recognition that are provided from the service providingdevice 120 of FIG. 1 through repeated experiences of the image displaydevice 100′ or mechanical learning. Based on this, the storage 420 canperiodically update the internal query information and delete unusedquery information.

The controller 430 controls the whole operation of the communicationinterface 400, the speech acquirer 410, the storage 420, the controller430, the display 440, the UI screen generator 450, and the speechrecognizer 460 in the image display device 100′. The controller 430provides the result of recognition for the speech waveform that isacquired by the speech acquirer 410 to the semantic informationgenerator 470 to obtain the semantic information and based on thesemantic information, searches for candidate queries includingcorresponding semantic information in the storage 420. Thereafter, thecontroller 430 provides the information associated with the searchedcandidate queries to the UI screen generator 450 to generate a list andthen displays the list on the display 440. Further, in the case of atouch screen type, the controller 430 obtains the search or answerinformation for the query selected by the user through communicationwith an external device through the communication interface 400 anddisplays the search or answer information on the display 440 again.

In addition, the controller 430 may perform an operation for imageprocessing. Since an image processing operation has been fully describedthrough the operation performer 310 of FIG. 3, further explanationthereof will be omitted.

Moreover, display 440 is similar to the display 320 of FIG. 3 andfurther explanation thereof is also omitted.

The UI screen generator 450 may generate a candidate query list based onthe query information for the candidate queries provided from thestorage 420 under the control of the controller 430. In response to therequest from the controller 430, the UI screen generator 450 may outputthe generated list.

The speech recognizer 460 provides the result of a recognitionassociated with the speech query acquired through the speech acquirer410. The speech recognizer 460 according to an embodiment of the presentdisclosure may be selectively driven. For example, if the speechrecognizer 460 of the image display device 100′ is configured to performrecognition without using the external service providing device 120 toobtain the result of recognition or if the conditions selected by theuser are satisfied, such as if an error occurs in the communicationnetwork 110 or the communication network 110 has severe loads, or if theinner loads of the service providing device 120 become severe, thespeech recognizer 460 may selectively operate.

When the image display device 100′ initially operates, there is noinformation for generating candidate queries, and thus the semanticinformation generator 470 may receive various types of recognitionresults from the external service providing device 120. The result ofrecognition may include candidate query information. The semanticinformation generator 470 may generate the semantic information usingthe provided result of the recognition and make the generated semanticinformation match a plurality of recognition results, i.e., queryinformation. In this process, the semantic information generator 470 mayseparate a word from the recognition result in the form of a sentence,add part of speech information through tagging, i.e., discriminating, ofthe part of speech and then generate index information. For example, theindex information may be a serial number associated with the semanticinformation. In an exemplary embodiment, if it is assumed that 32 piecesof semantic information are provided, the pieces of semantic informationmay be expressed by 32-bit data and each piece of bit information maycorrespond to the semantic information. In this case, if only two piecesof semantic information W1 and W3 exist, the index information may begenerated in a manner that first and third data of the 32-bit data aremarked “1”. If a query is input from an outside, the candidate queriesmay be searched by comparing index information. In addition, hashing maybe used as a method for searching for the candidate queries. Hashing isa calculation search method that calculates and finds the location wherea key exists using an arithmetic operation other than a search methodthrough comparison of the key values.

Referring to FIG. 5, the semantic information generator 470 may generatethe semantic information and provide information associated with thecandidate query that matches the generated semantic information to thecontroller 430. Then, the controller 430 sorts and stores thecorresponding contents in the storage 420. If the semantic informationgenerator 470 is integrated into the controller 430, it may becomepossible to directly sort and store the generated information in thestorage 420. However, the embodiment of the present disclosure may notbe specially limited thereto.

FIG. 6 is a block diagram illustrating the detailed structure of asemantic information generator of FIGS. 4 and 5 according to anembodiment of the present disclosure.

The semantic information generator 470 according to an embodiment of thepresent disclosure can store software in the structure as shown in FIGS.4 and 5, and can perform various operations through execution of suchsoftware. However, as illustrated in FIG. 6, the semantic informationgenerator 470 may operate by hardware as shown in FIG. 6. Of course,respective constituent elements may include programs.

From this viewpoint, the semantic information generator 470 of FIG. 6includes a part or the whole of a word separator 600, a part of speechtagger 610, and an index generator 620.

The term “include a part or the whole” is the same as that as describedabove and thus further explanation thereof will be omitted.

In an exemplary embodiment, if the word separator 600 is in the form ofa sentence and the text-based recognition result is input, words may beseparated based on, for example, spacing between words.

Then, the part of speech tagger 610 determines the part of speech of theword. For example, the part of speech tagger 610 determines words ofpredetermined parts of speech such as noun, adjective, and verb, andtags the part of speech on the word of the corresponding part of speech.

Thereafter, the index generator 620 generates index information to bematched to the input result of recognition, i.e., query, considering thecorresponding words as primary keys. Since this operation has beendescribed, further explanation thereof will be omitted.

FIG. 7 is a flowchart illustrating a process of driving an image displaydevice according to an embodiment of the present disclosure.

For convenience in explanation, referring to FIGS. 1 and 7, an imagedisplay device 100 according to an embodiment of the present disclosureacquires a speech query created by a user at operation S700. Thereafter,the image display device 100 may additionally perform a process forobtaining the result of recognition for the acquired query. The resultof the recognition may be obtained by executing an internal engine,i.e., a program, and may be obtained using an external free creationengine.

Further, the image display device 100 displays a query list thatincludes candidate queries having the same as or similar semantic as toa semantic of the acquired speech query at operation S710. In anexemplary embodiment, the semantic may be expressed by at least onepiece of semantic information.

Thereafter, the image display device 100 performs an operation relatedto the query that is selected from the query list that is displayed onthe screen at operation S720.

In an exemplary embodiment, referring again to FIG. 2, if a user creates“movie” 200 or “Let me know an amusing movie” 200, the image displaydevice 100 determines that the primary keyword is “movie”. The movie maybe the semantic. Further, the keyword “movie”, i.e., a candidate querythat matches the semantic information, “I will show you movie ranking.”220 may be provided to the user on the UI screen as a candidate query.Even if the image display device 100 does not pass through the processfor selection from the list, the image display device 100 may audiblynotify the user of the corresponding operation by default, and providethe result of performance to the user. Accordingly, the image displaydevice 100 may provide information on the movie ranking or movie times230 as the result of search.

FIG. 8 is a flowchart illustrating a process of driving an image displaydevice according to an embodiment of the present disclosure.

Comparing FIG. 8 with FIG. 7, an image display device 100 of FIG. 8acquires a speech query, and generates a query list that is composed ofcandidate queries having the same as or similar semantic to the semanticof the acquired speech query at operations S800 and S810. Such listgeneration using the candidate queries may be actually performed by theUI screen generator 450 of FIG. 4.

Then, the image display device 100 processes search or answerinformation in order to provide the search or answer informationassociated with the query selected from the query list to a user atoperation S820.

In an exemplary embodiment, FIG. 8 may be performed using anon-transitory computer readable recording medium, however the methodillustrated in FIG. 8 may be implemented using one or more programsstored on the non-transitory computer readable recording medium.

FIGS. 9A to 9C are diagrams illustrating a screen display method of animage display device according to an embodiment of the presentdisclosure.

Referring to FIGS. 1 and 9A to 9C, the image display device 100 presentsa query using a keyword related to a situation during execution of anapplication.

For example, if a user creates a query “today weather” or “weather”,candidate queries, such as “How's the weather today?” or “Will it raintoday?” may be displayed as shown in FIG. 9A.

Further, in the case where the user inputs a query, the image displaydevice 100 searches for similar queries, i.e., candidate queries, andpresents the searched queries to the user. If the searched query isselected, the image display device 100 responds through a relatedanswer, while if not, the image display device 100 sends the user'squery to the service providing device 120 of FIG. 1 to respond throughnatural language understanding.

Referring to FIG. 9B, if the user creates “movie recommendation”, theimage display device 100 may display “movie times” and “box officeranking” in addition to “movie recommendation” that is transmitted bythe service providing device 120 of FIG. 1.

The user may select the query from the list that is displayed on thescreen where the selection may be performed by touching the touch screentype screen or creating a speech.

Alternatively, the image display device 100 may operate in an offlinemode and show the corresponding result in accordance with the user'ssetting. Referring to FIG. 9C, if the image display device 100 is unableto access the Q/A service, the image display device may provide theservice through a Q/A list search that is stored at the image displaydevice 100. With respect to a query that is similar to the user's query,the image display device 100 searches for and presents the result of thesearch associated with the query in the image display device, and inother cases, the image display device 100 displays that there is noresponse. In an exemplary embodiment, as shown in FIG. 9C, an icon 900is displayed to indicate that the current Q/A service, i.e., the serviceby the service providing device 120 of FIG. 1 is not provided and/oravailable.

As described above, the image display device 100 of FIG. 1 according toan embodiment of the present disclosure may construct a personalized Q/Aset based on the user situation information (e.g., place and time), theuser profile, and past Q/A usage patterns. Through this, the candidatequeries as shown in FIG. 9C may be presented, and additionally requiredQ/A may be provided from the service providing device 120.

FIG. 10 is a flowchart illustrating a process of driving an imagedisplay device according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 10, at operation S1000, an inquiry is receivedfrom a user and at operation S1010, the image display device 100searches for a similar query stored in the storage a to determine if anyprevious query that is identified as similar to a previous query.

If the query selected by the user is not in the similar previous query,the image display device 100 provides a response to the user throughnatural language understanding (NLU) using the service providing device120 of FIG. 1 at operations S1020 and S1040.

When the query selected is similar to a previous query, the imagedisplay device 100 provides a response related to the search query tothe user at operation S1030.

Thereafter, the image display device 100 may store the user's query andthe response in the storage at operation S1050.

FIG. 11 is a diagram explaining a process of searching for a similarquery according to an embodiment of the present disclosure.

For convenience in explanation, referring to FIGS. 1 and 11, the imagedisplay device 100 according to an embodiment of the present disclosuremay perform a search for a similar query through two paths.

For example, if a query is initiated by the user, the image displaydevice 100 searches an internal Q/A storage device 1140 and obtainsinformation associated with the similar query as well as furtherinformation associated with the similar query that is personalized tothe user. If there is a plurality of information associated with thesimilar query, the image display device 100 may present the plurality ofinformation by the ranking in the form of a list to the user. Of course,it is not necessary to designate the ranking.

Further, if there is no information associated with the query, the imagedisplay device 100 may request information from the service providingdevice 120, receive information on the similarity query through the NLU,and generate a list to present to the user.

For example, the generation of the similar query information through thequestion understanding may be performed using a training samplegenerator 1100, a translation (or interpretation) possibility calculator1110, and a storage element 1140. The training sample generator 1100 mayextract a Q-A pair that is similar in semantic through model trainingwith respect to collection of input Q-A pairs. The translationpossibility calculator 1110 may calculate the translation possibilityfor the extracted Q-A pair. If the possibility is high as the result ofcalculation, the information is stored in a separate storage 1120, suchas a DB, in the form of a table.

Further, if there is no similar query that is searched through theinternal search of the image display device 100, a ranking arrangementoperator 1130 may search for the similar query from the informationstored in the service providing device 120 in the form of a table andprovide the searched similarity query to the user.

FIG. 12 is a block diagram illustrating another structure of an imagedisplay device of FIG. 1 according to an embodiment of the presentdisclosure.

Referring to FIG. 12, an image display device 100″ according to thisembodiment of the present disclosure may be, for example, a mobileterminal, and may include a part or the whole of a semantic searchengine 1200, a situation recognition manager 1210, and a personalizedQ/A storage device 1220. The term “include a part or the whole” has thesame meaning as that as described above.

The semantic search engine 1200 may serve to generate semanticinformation for a user's query.

The situation recognition manager 1210 may determine the user's currentsituation, and may search for the personalized Q/A storage device 1120based on the determined current situation and the generated semanticinformation. Accordingly, situation recognition manager 1210 may searchfor a similar query based on the user's situation (e.g., location, timeof day, etc.), even if the query of the same type as a previous query,and provide an answer to one query that is selected from similarprevious queries.

The personalized Q/A storage 1220 stores personalized similar queryinformation associated with the user. As described above, the similarityquery information may be stored in terms of a place in which the user islocated, current time when the query is initiated, a user profile, and apast Q/A usage pattern. Since this has been fully described, furtherexplanation thereof will be omitted.

FIG. 13 is a block diagram illustrating the structure of a serviceproviding device illustrated in FIG. 1 according to an embodiment of thepresent disclosure.

Referring to FIG. 13, the service providing device 120 according to anembodiment of the present disclosure may briefly include a query searchand responder 1300 and a query understanding and responder 1310.

The query search and responder 1300 includes a semantic search engine1301 and a Q/A storage device 1303. The semantic search engine and theQ/A storage device 1303 may perform operations to provide similarqueries to the user. For this, the semantic information associated withthe user's query is extracted, and the Q/A storage device 1303 issearched based on the extracted semantic information to provideinformation for the similarity query.

The query understanding and responder 1310 includes a part or the wholeof a natural language understanding device 1311, a natural languagegenerator 1313, a knowledge search engine 1315, and a knowledge storagedevice 1317. These constituent elements may serve to understand thequery selected by the user from the similarity queries and provide aresponse related to the understood query. For example, the naturallanguage understanding device 1311 may understand (or recognize) theselected query and the natural language generator 1313 may find akeyword through separation of the understood query by words. Theknowledge search engine 1315 may then search the knowledge storage 1317based on the keyword.

As described above, although it is explained that all constituentelements that constitute an embodiment of the present disclosure arecombined into one or operation in combination, it is not necessary thatthe present disclosure is limited to such an embodiment. That is, withinthe purpose range of the present disclosure, one or more of all theconstituent elements may be selectively combined to be operated.Further, although all the constituent elements can be implemented byindependent hardware, a part or the whole of the constituent elementsmay be selectively combined, and may be implemented as a computerprogram having program modules that perform a part or the whole of thefunctions that are combined in one or plural hardware configurations.Codes and code segments that constitute the computer program may beeasily induced by skilled in the art the present disclosure pertains.Such a computer program may be stored in a non-transitory computerreadable recording medium in the form of a program. In an exemplaryembodiment, the non-transitory computer readable medium is not a mediumthat stores data for a short period, such as a register, a cache, or amemory, but means a medium which semi-permanently stores data and isreadable by a device. Specifically, various applications and programs asdescribed above may be stored and provided in the non-transitorycomputer readable medium, such as, a compact disc (CD), a digitalversatile disc (DVD), a hard disc, a Blu-ray disc, a universal serialbus (USB), a memory card, and a read only memory (ROM).

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and detail may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. An image display device comprising: a display; acommunication interface; a memory configured to store a plurality ofquestions; and a processor configured to: acquire a speech queryassociated with a query uttered by a user, identify a plurality of wordsfrom the speech query, identify a primary keyword from the identifiedplurality of words, generate a query list composed of candidatequestions from a set of questions of the plurality of questions storedon the memory based on the identified primary keyword, control thedisplay to display the query list, and perform an operation related to acandidate question selected from the displayed query list, wherein theset of questions of the plurality of questions is selected from amongthe plurality of questions based on user situation information, whereinthe user situation information comprises one of a location of the useror a current time, and wherein the processor is further configured to,in response to identifying that there are no candidate questionscorresponding the speech query in the memory, provide the query listcomposed of candidate questions received from an external device throughthe communication interface.
 2. The image display device of claim 1,wherein the processor is further configured to: generate a query listinterface including the query list, and control the display to displaythe generated query list interface.
 3. The image display device of claim2, wherein the generating of the query list is further based on at leastone of a user profile or a past service usage pattern.
 4. The imagedisplay device of claim 1, wherein the processor is further configuredto recognize the acquired speech query, and wherein the generating ofthe query list is further based on the result of the recognition.
 5. Theimage display device of claim 1, wherein the communication interface isfurther configured to receive a result of recognizing the acquiredspeech query from an external device, and wherein the generating of thequery list is further based on the received result of the recognition.6. The image display device of claim 1, wherein the processor is furtherconfigured to: separate the query having a form of a sentence into theplurality of words, tag parts of speech of the separated plurality ofwords, classify the plurality of words based on predetermined parts ofspeech, and generate index information associated with the classifiedplurality of words based on the parts of speech, and wherein thecandidate questions are searched using the index information includingsame semantic information.
 7. The image display device of claim 6,wherein the query list is generated based on the classified words. 8.The image display device of claim 1, wherein the processor is furtherconfigured to provide search information or answer informationassociated with the selected candidate question.
 9. A method for drivingan image display device, the method comprising: acquiring a speech queryassociated with a query uttered by a user; identifying a plurality ofwords from the speech query; identifying a primary keyword from theidentified plurality of words; generating a query list composed ofcandidate questions from a set of questions of a plurality of questionsstored on the image display device based on the identified primarykeyword; displaying the query list; and performing an operation relatedto a candidate question selected from the displayed query list, whereinthe set of questions of the plurality of questions is selected fromamong the plurality of questions based on user situation information,wherein the user situation information comprises one of a location ofthe user or a current time, and wherein, in response to identifying thatthere are no candidate questions corresponding the speech query in aninternal memory, the method further comprises providing the query listcomposed of candidate questions received from an external device. 10.The method of claim 9, further comprising: generating a query listinterface including the query list, wherein the displaying of the querylist comprises displaying the generated query list interface.
 11. Themethod of claim 10, wherein the generating of the query list is furtherbased on at least one of a user profile or a past service usage pattern.12. The method of claim 9, further comprising: recognizing the acquiredspeech query, wherein the generating of the query list is further basedon the result of the recognition.
 13. The method of claim 9, furthercomprising: receiving a result of recognizing the acquired speech queryfrom an external device, wherein the generating of the query list isfurther based on the received result of the recognition.
 14. The methodof claim 9, further comprising: separating the query having a form of asentence into the plurality of words; tagging parts of speech of theseparated plurality of words; classifying the plurality of words basedon predetermined parts of speech; and generating index informationassociated with the classified plurality of words based on the parts ofspeech, and wherein the candidate questions are searched using the indexinformation including same semantic information.
 15. The method of claim14, wherein the query list is generated based on the classified words.16. The method of claim 9, wherein the performing of the operationcomprises performing an operation to provide search information oranswer information associated with the selected candidate question. 17.A non-transitory computer readable recording medium including a programthat, when executed by a processor, causes the processor to control for:acquiring a speech query associated with a query uttered by a user;identifying a plurality of words from the speech query; identifying aprimary keyword from the identified plurality of words; generating aquery list composed of candidate questions from a set of questions of aplurality of questions stored on an image display device including thenon-transitory computer readable recording medium based on theidentified primary keyword; displaying the query list; and performing anoperation related to a candidate question selected from the displayedquery list, wherein the set of questions of the plurality of questionsis selected from among the plurality of questions based on usersituation information, wherein the user situation information comprisesone of a location of the user or a current time, and wherein theprogram, when executed by a processor, further causes the processor tocontrol for, in response to identifying that there are no candidatequestions corresponding the speech query in an internal memory,providing the query list composed of candidate questions received froman external device.
 18. The non-transitory computer readable recordingmedium of claim 17, wherein the generating of the query list is furtherbased on at least one of a user profile or a past service usage pattern.